Repository navigation
Conversation
|
R: @ahmedabu98 |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
| + " determined; a '**' can only come after the last placeholder.", | ||
| pathPattern); | ||
| } | ||
| } |
There was a problem hiding this comment.
Maybe this is too strict? The following template should be valid:
gs://exports/**/{table}/*.parquet
i.e. if the placeholder is the lowest directory
There was a problem hiding this comment.
Relaxed a bit so there can be at most one **.
There was a problem hiding this comment.
But now that would reject something like gs://bucket/{table}/**/dir/**/*
There was a problem hiding this comment.
These rules are to identify the table membership/name from the {placeholder} parameter. I think we can assume that incoming files represent the datafiles that need to be processed . If we are not able to parse the placeholder from a file then it should be an error. User can just do gs://bucket/{table}/** to achieve the same.
MatchAllFiles pattern can be used to filter initial set of files.
I can try find a way to validate pattern with 2+ double star if you feel strongly though.
fb03152 to
8f23d53
Compare
Adds the two self-contained pieces that dynamic destinations for
IcebergIO.AddFilesarebuilt on: a path-pattern matcher, and the
DoFnthat uses it to name a destination tablefrom each file's path.
AddFilesregisters every input file into one table named bytable. Migrating a lakeusually means a tree of per-table folders, which today needs one
AddFilesper table.The goal is for one transform to route each file to the table its path names:
This PR adds only the routing primitives so they can be reviewed on their own. The
pre-pass, registration and schema-transform surface follow separately.
PathPatternA whole-path glob with
{name}placeholders, each capturing exactly one path segment.Globbing is Beam's own (
FileSystemUtils.wildcardToRegexp), so*,**and?behave asthey do in any other Beam file pattern; placeholders are translated to capturing groups
between the globbed fixed parts.
The whole path must match, so the pattern decides both which table a file belongs to and
whether it belongs at all.
compilerejects patterns that cannot answer thatunambiguously:
{a}twice{a b},{a.b}gs://lake/**/{tbl}/****above a placeholder leaves the captured folder undeterminedgs://lake/{tbl}//, so it names a folder and matches no filerender(template, captures)fills the table template, andcheckRenderable(template)refuses a template naming a placeholder the pattern never captures, so
table: default.{db}against.../{tbl}/**fails at construction rather than producing one errorrow per file.
RouteFilesDoFn<String, KV<String, String>>with two outputs:ROUTED:KV<table, path>for a file the pattern matchedERRORS: a{file, error, table}row for a path it did not, withtableempty becausethere is no table to attribute it to
An unmatched path becomes an error row rather than being silently dropped: a mistyped
pattern must be visible.
PathPattern.compileandcheckRenderableboth run in theconstructor, so a bad pattern or template fails at pipeline construction instead of once
per file on a worker.
RouteFiles.ERROR_SCHEMAis declared locally here. When the transform is wired up itbecomes the same schema as
AddFiles' error output, so the two can be flattened; thatchange belongs with the wiring.
**/*to**, which Python'sFileSystem.translate_patterndoes not. Relevant only if youcompare a
path_patternagainst a glob written for Python'sMatchFilesupstream.